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ABSTRACT 

An external evaluation was made of the Competitive 
Skills Project, a National Workplace Literacy Program carried out in 
partnership between El Camino College and BP Chemicals. Among the 
problems identified were the following: (1) because the original 
director and his successor left the project, the original evaluation 
design could not be implemented; (2) the sponsor precluded testing of 
all participants; and (3) lack of a coordinator and of teacher 
training resulted in failure to collect baseline data on participants 
against which learner progress could be measured. The external 
evaluation included technical assistance at the beginning of the 
project, group discussion with supervisors and managers, and focus 
groups with workers, managers, and supervisors. Major outcomes 
included the following: (1) improved skills; (2) positive workplace 
climate and increased worker confidence; (3) greater educational 
opportunities; and (4) increased productivity. Success of the program 
was attributed to flexible approach, responsiveness to client needs, 
and participation of all partners in devel opment . Concerns expr es s ed 
included a need for smaller classes, slower pace, and instruction in 
basic communication skills that were not job specific. The following 
recommendations were made: (1) develop a conceptual framework that 
helps to operationalize the curriculum; (2) develop an assessment 
framework that operates on two levels, program wide and course 
specific; (3) respond to the needs of workers who need additional 
support in order to develop the necessary total quality management 
skills; and (4) consider offering a series of workshops on 
communication strategies for total quality management to help build 
cross-cultural understanding and facilitate communication between 
front-line workers and supervisors. (CML) 
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PART I: CONTEXT AND CHALLENGES 



The Competitive Skills Project (CSP) was carried out through a partnership between El 
Camino College, a public community college serving 27 000 students, and BP Chemicals 
(HITCO), a manufacturing firm, specializing in high temperature advance composite 
products for aerospace and industrial markets. Hitco has a highly diverse work force and 
many workers do not have the basic skills necessary to participate effectively in the Total 
Quality Management approach the company is implementing. The technology changes that 
are imminent (Continuous Improvement, Statistical Process Control) also require skills 
upgrading. In response to company needs, tl * Competitive Skills Project was established 
in July, 1990 and concluded i 1 April, 1992. A no cost extension was granted until July 31, 
1992. 

There are many indicators that the project was successful overall (see documentation by 
project staff) and resulted in positive outcomes for the company and its employees (see 
below). However, as is the case with many workplace projects, CSP faced a number of 
challenges that made learner assessment and program evaluation difficult. Since both the 
original director (Nick Kremer) and his successor (Georganna Ahlfors) left the project, the 
original evaluation design could not be implemented. In addition, HITCO's concerns about 
worker testing precluded the use of a standardized test for all participants. Since the project 
did not have a curriculum coordinator, and teachers were not trained in assessment 
techniques, there is no data that would allow for comparisons across classes. (However, 
each teacher administered class tests and graded participants). As a result of these 
difficulties in implementation, the project was not able to collect base line data on 
participants against which learner progress could be measured. In the absence of such data, 
it is not possible to aggregate data across learner groups or determine what percentage of 
employees reached the objectives outlined in the original evaluation plan. 
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PART II: LEARNER ASSESSMENT AND 
PROGRAM EVALUATION 



The success of the project can be judged by two types of data: (1) quantitative data on 
levels of participation and number of courses offered and (2) qualitative data on learner 
outcomes (selected participants) and workplace impacts (reports from supervisors and 
managers). Quantitative data, along with program implementation data and reports on goal 
achievement, can be found in the program final report submitted by El Camino directly. 
Qualitative data focusing on program outcomes as perceived by selected participants and 
stakeholders is addressed in this report. 

A. METHODS 

Assessment and evaluation activities consisted of two components: (1) internal assessment 
conducted by project staff and (2) external evaluation conducted by the outside evaluator. 

1. Internal evaluation consisted of the following activities: 

a. A needs assessment and literacy audit of supervisors and managers 

b. Informal assessment by teachers to place learners into appropriate classes and 
design curriculum that meets their needs 

c. Ongoing assessment and testing by teachers on the content covered in the 
various classes; final test 

d. Class evaluations by participants indicating degree of satisfaction with the 
instruction offered 



e. Supervisor surveys to measure satisfaction with the instruction and gauge the 
effectiveness of the program 

(For outcomes on these activities, see Project Final Report submitted by El 
Camino). 

External evaluation included: 

1. Technical assistance at the beginning of the project (with Mr. Kremer and Ms. 
Ahlfors) to discuss strategies for implementing ongoing learner assessment as 
part of a comprehensive evaluation design. 

Outcomes: The possibility of using standardized literacy tests such as the 
ETS test of Applied Literacy or the BEST test were discussed 
and a broad assessment framework was outlined. Alternative 
assessments such as learner portfolios, simulations, case studies 
and performance tests were discussed as well. The evaluator 
suggested that surveys and focus groups be used to judge the 
impact of the program on employee performance. 

2. Group discussion with several HITCO supervisors and managers to discuss 
the purpose of evaluation and identify concerns. 

Outcomes: HITCO personnel stressed the importance of "problem solving" 
and "reasoning skills" required for Total Quality management. 
They also emphasized the importance of communication skills 
for employees who are not fully proficient in English. 



Formative evaluation 

Outcomes: Due to the challenges presented by the changes in directorship, 
the program was not able to implement a formative evaluation. 
Neither was it possible to carry out a comprehensive summative 
evaluation since outcome data was not comparable across 
groups. 

Final evaluation based on focus groups with workers, managers, and 
supervisors to document HITCO's perception of the impact of the program on 
workers' lives, productivity and company morale. 

The worker group consisted of seven employee who had participated in the 
program. These participants had taken DOE funded classes in Vocational 
English as A Second, English communication skills, and Math for Quality. 
The supervisor group included six individuals each of whom supervises one of 
the employees interviewed in the participant focus group. The management 
group included managers in the areas of Manufacturing, Plant management, 
Human Resources, and Quality Assurance as well as the Vice President and 
General Manager of Hitco and the Vice President of Quality Assurance. 
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B. MAJOR FINDINGS 



"The more you know, the greater the chance that they will keep you on " 

Employee who participated in the Competitive Skills Project 

/• Improved Skills 

All respondents cited positive outcomes for the project. Workers, in particular, stressed that 
the project helped them to be prepared for job changes since manufacturing jobs are 
becoming increasingly competitive. They saw the basic skills program as an opportunity to 
gain more education and upgrade their skills. Bilingual employees said that improved 
English and greater communication skills gave them a chance for job mobility. Workers also 
stressed that the classes prepared them to work in teams, a required skill for Total Quality 
Management. 

The workers who had participated in the project indicated that their skill levels had 
improved, both in English and in Math. Participants from the ESL classes told the evaluator 
that they were better able to understand what goes on at work (and why) and found it easier 
to understand supervisors and co-workers. 

All respondents reported that project participant had acquired the kind of skills needed for 
effective TQM operations. Supervisors indicated that many of the employees who had 
difficulty talking in front of the group showed improvement. Continuous exposure to English 
was seen as necessary so that they could effectively participate in quality teams. 
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2. Positive workplace climate and increased worker confidence 

Supervisor and managers also mentioned that the overall climate of the workplace had 
improved. By introducing classes and offering release time, management was seen as 
"people who cared", as a result, productivity increased since employees were willing to 
work harder and work more efficiently. The project thus was seen as an essential 
component in the companies attempt to develop the skills and abilities of all employees: 
workers, supervisors, and managers. 

Supervisors reported that employees appeared proud of their new found skills and showed 
greater confidence in their ability to communicate with others. The managers in particular 
indicated that employees had made great strides in expressing themselves. Some had spoken 
up for the first time at quality meetings and interaction between supervisors and employees 
had improved, as well. Since open communication has been identified as a critical factor in 
continuous improvement and collaboration between management and labor, these changes 
were seen as particularly significant. 

The project has an a positive impact on the lives of workers beyond the workplace. 
Program participants and their supervisors reported that increases in skills helped employees 
feel more confident in their ability to deal with everyday communication and math problems. 
Being able to help children with their homework was seen as an important side benefit of the 
training. 

3. Greater educational opportunities 

One supervisors also felt that offering classes at the work site (instead of "across the street", 
(at the college, where for credit classes are offered) provided additional benefits for 
participants who did not feel confident enough to take "college courses". In the past, some 
employees had found the college quite daunting and became discouraged. They then dropped 



out, "costing another employee a seat" and wasting taxpayers money. This supervisor 
praised the program for allowing employees "to get their feet wet" and giving them the 
opportunity to participate in an educational program that is supportive and encouraging. 

The fact that the project offered a variety of classes that involved various types of problem 
solving was also seen as a positive factor. Managers pointed out that workers now seemed 
better prepared to deal with "cross-training", an essential part of TQM. The project was 
seen as contributing to both the "soft" and "hard" factors of TQM. Soft factors included 
better communication skills, greater willingness to work together, and increased self- 
confidence. Hard factors included higher levels of quantitative literacy and computer 
literacy. The managers interviewed indicated that program participants had improved in all 
of these areas. 

4. Factors associated with the success of the program 

The managers interviewed indicated that much of the success of the project was due to three 
factors: (1) flexibility in approach; (2) responsiveness to the needs of the client and (3) 
participation of all partners in the development of the project. The group offered these 
details: Unlike traditional skills classes offered at the college, , the functional context 
instruction of CBS was based on an assessment of company needs and supervisor concern. 
There were few barriers to participation, and educational guidance was provided to help 
employees with goal setting. The fact that managers were involved in the skills analysis and 
were kept up to date on curriculum development was also seen as an important factor in the 
effectiveness of the program. . 

The collaborative nature of the project was seen as fitting well within the framework for 
change that the company supports. Managers mentioned that attitudes that maintain that 
"people on the shop floor don't know anything" are changing. Line workers now want to 
participate to hear what is going on, to have their voices heard. 



5. Increased productivity 

Both supervisors and managers indicated that the quality of work has gone up and the amount 
of work that needs "rework" went down. They also reported a greater rate of "no defects" 
and a decrease of "operator induced" errors (exact data is not available). 

Although it is not possible to establish a one-to-one relationship between the literacy 
program and improved workplace performance, supervisors and managers felt that there was 
a strong correlation between program participation and increased productivity improvement 
can be established 

6. Concerns expressed by workers, supewisors, and managers 

Both supervisors and workers felt that smaller class sizes were needed. There was a 
particular concern about the progress that employees with lower levels of proficiency were 
able to make. According to the supervisor and the employees who participated in the 
communication skills classes, the instruction moved a bit too fast for participants who had 
only minimal English skills. (Similar concerns were expressed about the math class which 
moved too fast for some of the learners). 

In addition, several of the employees wished for more opportunities to learn basic functional 
communication skills (i.e. asking for information, explaining a problem; making a 
suggestion) that were not necessarily tied to job performance. These participants said the 
had hoped that the program would ' elp them deal with every day communication problems, 
on the job and off. Several supervisors felt that the greatest need for English was at the 
most basic levels of communication and suggested that "practical English" and 
"conversational ability" should be stressed. In their view, job specific technical knowledge 
can be introduced later, after employees have gained fluency in English. 
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PART HI: COMMENTS AND RECOMMENDATIONS 



The challenges that the project faced and the successes it experienced reflect many of the 
difficulties that workplace literacy field as a whole has experienced (U.S. Department of 
Education, 1 1992). The lack of stability in personnel makes it difficult to implement a 
coherent framework and insufficient assessment experience on part of the staff results in 
uneven evaluation data. This state of affairs is true for the entire adult literacy field 
(Alamprese, 1988 2 , Wrigley and Guth, 3 1992). 

On the positive side, the project's success in increasing the communication and math skills of 
employees while at the same time boosting their confidence, mirrors the results of other 
innovative programs, as well. This particular project has been successful in increasing the 
skills required for Total Quality Management, thus helping to reshape the company and 
contributing to its economic success. By operating a technology lab, the project also 
provided access to the kind of technology knowledge that will be required in the future. 

Another noteworthy outcome of the project is the change in organizational climate that it 
helped effect. By facilitating communication among employees as well as between 
supervisors and front-line workers, the project has had an impact on the overall workplace 
that goes beyond the individual who attended classes. In addition, the project has led to a 
greater appreciation cf the company's educational efforts, and workers seem ready to 
contribute additional efforts to the company's success. 



Workplace Literacy: Reshaping the American Workforce:U.S. Department of Education, Office of 
Vocational and Adult Education; Washington, D.C., 1992 

2 Adult literacy: Research and Development: An Agenda for Action by Judith Aleinprese; Southport 
Institute for Policy Analysis, 1988 

3 Bringing Literacy to Life: Issues and Options in Adult ESL Literacy by Heide Spruck Wrigley and 
Gloria J. A. Guth; San Mateo, California; Apuirre International, 1992 
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The project has proven to be a stimulus to other changes at Hitco as well. In the spring of 
'92, Hitco conducted a wellness fair for its employees. It also held a "town meeting" where 
the economy and the company's role in the international market place were discussed, and 
employees and management engaged in a dialogue about the company's future. Managers 
stressed the link between these new projects and the workplace literacy program. The 
project has thus contributed to a collaborative model of worker education, where all partners 
participate and share responsibilities. 

Recommendations: 

Although a summative evaluation for the 1990/1992 cycle, this evaluation serves a formative 
function as well, since a new project, The Basic Skills For Quality Project, has been funded 
for the next cycle. Staff should consider the following: 

1. Develop a conceptual framework that helps to operationalize the curriculum. This 
framework should outline the overall educational approach and list the kind of skills that are 
taught across components (e.g., problem solving, communication, effective reasoning, team 
work) as ). In addition, the project should link the overall goals to ;he specific skills that are 
being taught in each course (e.g., math for quality, functional ESL, computer accessing 
skills). At present, the curriculum appears somewhat fragmented and disjointed. 

2. Develop an assessment framework that operates on two levels: program-wide and 
course-specific. 

a. Program wide assessment should include strategies for determining needs, 
assessing proficiency levels to that learners can be placed correctly and 
measuring the progress that learners make as they participate in the program. 
It should also include means for capturing the impact the project is having on 
the overall workplace (increased production, team building, better 
communication). Program-wide assessment requires that the program collect 
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base line data at the beginning of each cycle against which progress and 
achievement can be measured. Program wide assessment should include input 
and feedback from all stakeholders: teachers, learners, supervisors, 
management. Appropriate tools include interviews, surveys, observations and 
focus groups. (A major shortcoming of the present project was that no 
program wide assessment was conducted). 



b. Course-specific assessment should include assessing learner's knowledge and 
skills at the beginning of each course, at midpoints, at the end of a course. 
Assessment techniques should focus on performance-based assessments, such 
as portfolios, case studies, simulations, and role plays, rather than traditional 
pencil and paper tests. Observation charts, developmental checklists or 
competency lists can also be used. Strategies for assessing the success of the 
instructional lab should be included as well. (The present project relied to 
heavily on teacher-made, pencil and paper assessments). 

Both the conceptual framework and the assessment framework need to be incorporated into a 
comprehensive evaluation plan. While the development of an overall framework should be 
the responsibility of a curriculum coordinator, individual teachers may need training in 
developing assessments suitable for the workplace. (At present, the assessments used in 
classes don't show a strong enough relationship to the demands of the actual workplace). 

3. Respond to the needs of workers who need additional support in order to develop the 
necessary TQM skills. Consider two options: 



a. Split a course in two - one level for low intermediate - another for high 
intermediate; the low intermediate class should focus on the kind of functional 
communication and problem solving skills needed in a TQM environment. 
Low intermediates should be allowed to take several classes in a course 
sequence. (At the lower levels of proficiency, a single course does not 
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provide enough opportunity for lasting skill development.) 



b. Use teachers who know how to set up collaborative learning activities that 
work well in a multi-level classroom and provide additional support in the 
form of tutoring or focused Learning Center activities. 



4. Consider offering a series of workshops on "Communication Strategies for TQM" to 
help build cross-cultural understanding and facilitate communication between front- 
line workers and supervisors. In such workshops, staff can discuss the purpose of the 
program and explain the nature of language, literacy, and learning in the workplace. 



By developing a conceptual framework and linking it to a comprehensive assessment model, 
the project can develop assessment data that is comparable across groups and can provide 
strong documentation of success in the future. Implementing extended classes and 
suggesting supervisor workshops helps the program to remain responsive to learner concerns. 
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